ControlMM is an innovative technical framework developed jointly by the Chinese University of Hong Kong and Tencent, aimed at addressing challenges in multi-modal full-body motion generation. This framework supports multi-modal inputs from text, speech, and music to generate full-body motions that match the content. It employs the ControlMM-Attn module to process dynamic and static human topology in parallel, achieving efficient movement knowledge learning. A phased training strategy is adopted, progressing from text to motion pre-training, then to multi-modal control adaptation, ensuring the model's effectiveness under various conditions.

A research team from The Chinese University of Hong Kong and the Chinese Academy of Sciences has proposed a full modal pre-training paradigm called MiCo (Multimodal Context). This method has achieved significant results in the field of multimodal learning, setting new state-of-the-art (SOTA) records in 37 categories.